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ABSTRACT 



Media reports are an important source of new scientific 
knowledge, and this information can affect the decision making process. This 
study explores how formal educational settings prepare students to evaluate 
informal scientific information. Participants in this study came from 
different science backgrounds, including adults who did not study post 
secondary studies and university students from different majors. The 
differences among the participants were investigated. (YDS) 
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Objectives and Theoretical Underpinnings 

Media reports of scientific research are pervasive and a potentially important source of new scientific knowledge, not only for students 
who pursue careers in science-related fields but also for the general public (Wellington, 1991). Citizens in a democratic society can be 
called upon to contribute to policy issues such as acid rain or AIDS, and in doing so, they may require the skills necessary to critically 
evaluate media reports of scientific research. Furthermore, personal decisions, such as choice of medical treatment, can be influenced by 
information presented in the media. The prevalence of news reports and the importance of decisions based on scientific research 
presented in these reports makes evaluative skills an important component of science literacy and a valued outcome of contemporary 
science education (Glynn & Muth, 1994). 

Researchers are showing increasing interest in the informal opportunities to learn about science afforded to the public by the media. Furthermore, 
research is being conducted on the skills necessary to evaluate scientific news reports critically (e.g., Korpan, Bisanz, Bisanz, & Henderson, 1997; 
Norris & Phillips, 1994). We have initiated research on how students read and evaluate media-style reports about scientific research. One goal is to 
describe developmental changes and individual differences in knowledge structures that can be represented in terms of constituent elements, relations 
among those elements, and patterns of activation. Theories of knowledge structures have played a major role in research on how information is 
represented and used in many domains (e.g., Chi, 1992) and are essential for improving instruction. In this study, we asked participants with 
differing numbers of science courses to evaluate brief media-style reports of scientific research. This research will help us characterize how knowledge 
acquired within formal educational settings prepares students to deal with scientific information presented in informal contexts. 

A second goal of our research was to determine how evaluation of scientific news briefs is influenced by text characteristics. In a previous study 
(Korpan et al, 1997), we found tentative evidence that students! evaluation is influenced by differences along two dimensions: typicality, the degree to 
which the domain of the media report is related to phenomena normally studied in the science curriculum for university-bound students (e.g., biology, 
chemistry, physics); and plausibility ,, the degree to which conclusions stated in the reports are credible. In the present study, we systematically 
manipulated typicality and plausibility to determine the extent to which these dimensions influence the type of questions participants ask when they 
evaluated news briefs. 

Participants 
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Four groups of adults with differing levels of science education participated in the study, adults who completed high school but had not pursued 
post-secondary studies at the time of the study (i.c., non-university participants, n = 34); first-year university students (n = 24); fourth-year university 
students majoring in English (n = 24); and fourth-year university students majoring in Psychology (n = 24). All university participants began their 
university program within a year of high school graduation. The age range of the adults who did not pursue post-secondary education matched the age 
range of the entire university sample. As indicated in Table 1, adults who pursued postsecondary studies have more science background at the high 
school level than those who did not go on to university. Among the three groups of university students, there were no significant differences in the 
number of science courses completed at the high school level. At the university level, however, Psychology majors had completed approximately six 
times as many science-related courses (i.c., the sum of university level courses completed in math & statistics, natural & physical sciences, and social 
sciences) as first-year students and English majors. 



Table 1 

Science Background as a Function of Educational Level 
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No University 


l st -year University 
Students 


4 lh -Year English 
Majors 


4 th -Year Psychology 
Majors 


Semesters of High School 
Math 


3.1 3 


4.83 


4.33 


3.96 


Semesters of High School 
Science 


6.38 


1 2.17 


10.71 


11.33 


University Math & 
Statistics Courses 


0 


.88 


.46 


2.69 


University-Natural & 
Physical Sciences Courses 


0 


2.04 


1.58 


3.63 


Uni versity -Social Science 
Courses 


0 


1.54 


.96 


18.12 


Total # of Courses 


9.5 1 


21.46 


18.04 


39.73 



Materials and Methods 



Each participant read media-style news briefs about scientific research in four domains, each having the following format: (a) a general concern or 
issue is described; (b) researchers report a finding; and (c) an independent group concludes that the finding is important for addressing the general 
concern or issue. The four news briefs represented orthogonal combinations of typicality (High, Low) and plausibility (High, Low). The news briefs 
are provided in the appendix. All students rated the typicality and plausibility of the conclusion reported in each news brief using a 7-point Likert 
scale. For Plausibility , the question was: How likely do you think it is that the reported conclusion is true? 1 = very unlikely, 7 = very likely. For 
Typicality , the questions was: How closely related is the conclusion to topics covered in the natural and physical sciences such as Physics, Chemistry, 
or Biology? 1 = very unrelated, 7 = very related. Their ratings confirmed our method of classifying news briefs and did not differ across the four 
groups. 

Students were also asked to list questions that needed to be answered for them to decide whether the conclusion reported in each news brief was true 
and to justify their questions by explaining how knowing the answer to eaeh question would help them determine whether the reported conclusion is 
true. 

We used a comprehensive taxonomy developed by Korpan et al. (1994) to classify the questions and justifications. Only questions are discussed in this 
paper. The major topics in the taxonomy used to classify questions are described below. 

1. Social Context includes questions about the people, institutions, and publication outlets associated with the research. These 
questions reflect an interest in issues of credentials, prestige, and bias related to who conducted, evaluated or promoted the 
research and where the research was conducted. 

2. Theory includes questions about the properties of the treatments and mechanisms underlying the treatment effects reported 
in the news briefs. These questions reflect an interest in why the reported effects might have occurred. 

3. Method includes questions about research design, procedures, subject selection, and measures. These questions reflect him 
the research was conducted. 

4. Data includes questions about the data used to make the conclusions and about the statistical methods used to analyze or 
interpret the data. These questions reflect an interest in what was observed in the reported studies. 

5. Related Research includes questions about findings from other relevant scientific studies. These questions reflect an interest 
in issues of replication and consensus. 

6. Relevance includes questions about the importance, applicability, or generalizability of the findings or about the impact of 
the research. These questions do not help determine the truth of the conclusions, however. 

Ratings and question-generation data were collected initially from university students and later from non-university participants. For each news 
brief, university students were given 20 minutes to rate the conclusion on four dimensions (plausibility, typicality, knowledge, and interest), justify 
their ratings, write down their questions, and justify their questions. Non-university participants were given 8 minutes to provide their ratings, 
generate questions, and justify their questions. They were not asked to justify their ratings. The time allocated to each of these tasks was determined 
by the participants. 

Selected Results and Discussion 



Group Effects 

The first goal of this research was to determine how the range of questions generated and the salience of particular topics are affected by background 
in science. We investigated educational effects in a number of ways. First, we looked at how groups differed in terms of number of questions generated 
with regard to each topic across the four news briefs (Sec Table 2: Mean Number of Questions Generated as a Function of Educational Level and 
Topic). To control for possible differences in the time allocated to the question-generation part of the task, we also analyzed group differences in terms 
of proportions (see Table 3: Mean Proportion of Questions Generated as a Function of Educational Level and Topic). For example, Social Context was 
calculated by dividing the number of questions generated about social context across the four news briefs by the total number of questions generated. 
We also investigated the first two questions that participants generate when they evaluated all four news briefs (see Table 4) and the percentage of 
individuals who generated at least one question about eaeh topic for all four news briefs (see Table 5). All four tables support the following trends. 
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y 1. Across all 4 groups and 4 news briefs, questions were focused primarily on how the research was conducted (Method) and why the reported results 
V occurred (Theory). These topics tend to be the most salient to participants, almost to the exclusion of other topics. In science, conclusions are closely 
linked to method and theory. Historically, concerns that students might not recognize these links may have resulted in an advocacy for 
process-oriented approaches to science education. The prevalence of questions about method and theory may reflect this approach in science 
classrooms. 

2. Psychology majors generated more questions than the other 3 groups, with most of their questions focused on Method. Furthermore, First-year 
university students and English majors focused more of their questions on Method than those who did not continue on to university. This trend may 
be associated with differences in the number of science courses completed (See Table 1) and, therefore, differences in exposure to process-oriented 
approaches to teaching science (DeBoar, 1991). 

3. Encouragingly, all groups demonstrated a high level of interest in why the reported results occurred by generating several questions about Theory. 
One possible explanation for this Finding is that the amount of formal science education that most citizens attain in high school is sufficient to 
stimulate interest in theory and an awareness of its importance. Another possibility is that informal learning experiences may acquaint individuals 
with this aspect of science. Because media reports about science are pervasive, research focused on the nature of these reports is necessary to 
understand the role they play in informal learning and critical thinking about theory and other aspects of scientific activity (e.g., Zimmerman et al., in 
preparation). Other types of informal learning (e.g., visits to museums) may also play a role in familiarizing individuals with science and should be 
studied further (e.g., Dierking & Martin, 1997; Korpan et al., 1997). 

Also noteworthy is the Finding that non-university participants demonstrated more interest in Theory than university students. One possible 
explanation for this Finding is that they have little familiarity with other issues, and therefore tend to focus on mechanisms underlying the reported 
effect. 

4. Fourth-year Psychology and English majors directed more of their questions to issues concerning Social Context than the other two groups of 
participants. As Social Context issues are usually not emphasized in science classrooms, even at the undergraduate level, it is possible that students in 
both disciplines acquired this type of knowledge primarily through their general university experiences with various research communities. 

Questions about social context were not varied, however. Questions were directed at who cthe researchers were and their qualifications, which reflects 
an awareness that scientific research takes place within a research community that can influence the selection of research questions and the 
interpretation and acceptability of results. Questions about where the research was conducted and how the research was funded were rarely asked by 
university participants, and never asked by non-university participants. 

5. Disappointingly, all groups demonstrated a comparably low level of interest in quantitative aspects about what was found in the reported studies 
(Data) and how it is related to extant research in the scientific literature (Related Research). This finding was especially surprising for Psychology 
majors, given their course work and demonstrated ability to ask numerous sophisticated questions about Method. It seems that students, even those 
who have completed several science-related courses at the university level, need explicit encouragement to think about a variety of scientific issues 
when evaluating the credibility of research. 



Table 2 



Mean Number of Questions as a Function of Educational Level and Topic 





No University 


1 st Year 
U n i versity 
Stu dents 


4 1 * 1 Year English 
Majors 


4 1 * 1 Year Psychology 
Majors 


Average Across 
Education Level 


Social Context 


1.52 


1 .38 


2.88 


3.33 


2.28 


Theory 


5.11 


4.50 


4.33 


4.00 


4.49 


Methods 


2.88 


5.92 


5.46 


9.46 


5.93 


Data 


1 .62 


1 .88 


1.33 


2.13 


1.74 


Related Research 


. 3 5 


.21 


.21 


.46 


.31 


Relevance 


2.47 


1.79 


2.08 


1.88 


2.06 


Total # of Questions 


13.95 


15.68 


16.29 


21.26 


16.80 



Table 3 



Mean Proportion of Questions as a Function of Educational Level and Topic 
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No University 


1 st Year University 
Students 


4 th Year English 
M ajors 


4 th Year Psychology 
Majors 


Average Across 
Education Level 


Social Context 


. 1 1 


.09 


. 1 8 


. 1 6 


. 1 4 


Theory 


.37 


.29 


.27 


. 1 9 


.27 


Methods 


. 2 1 


.3 8 


.34 


.44 


.35 


Data 


. 1 2 


. 1 2 


.08 


. 1 0 


. 1 0 


Related Research 


.03 


. 0 1 


.01 


.02 


.02 


Relevance 


. 1 8 


. 1 1 


. 1 3 


.09 


. 1 3 



Table 4 

Mean Number of Times Each Topic is Mentioned in the First Two Questions 
of all Four News Briefs (8 Opportunities) as a Function of Group 





No University 


1 st Year University 
Stude nts 


4*^ Year English 
Majors 


4 th Year Psychology 
Majors 


Average Across 
Education Level 


Social Context 


0.7 9 


0.5 


1.42 


1 .5 


1 .02 


Theory 


3.26 


2.83 


2.04 


1.79 


2.56 


Methods 


1.4 1 


2.79 


2.75 


3.25 


2.44 


Data/Statistics 


0.4 1 


0.79 


0.8 3 


0.75 


0.67 


Related Research 


0.12 


0.04 


0.08 


0.04 


0.08 


Relevance 


0.82 


0.33 


0.75 


0.42 


0.6 



Table 5 

Percentage of Individuals Makings at Least One Request 
for a Given Topic on All Four News Briefs 





Social Context 


Agent/ Theory 


Methods 


Data/Statistics 


Related Research 


Relevance 


No University 


6 


5 0 


9 


3 


0 


6 


1 st Year University 


8 


2 5 


29 


4 


0 


0 


4*^ Year Psychology 
Majors 


25 


3 8 


6 7 


8 


0 


4 


4 1 ^ Year English 
Majors 


29 


2 5 


25 


0 


0 


0 



News Brief Effects 
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The second goal of this research was to determine how the types of questions participants generated was affected by the text dimensions of Plausibility 
and Typicality . This was addressed by investigating the mean number of questions generated for each topic as a function of news brief (See Table 7). 
Two notable trends are discussed. 

6. All four groups generated more questions for news briefs rated low in plausibility than news briefs rated highly plausible. Furthermore, the 
emphasis was on who conducted the research (Social Context) and the mechanism underlying the reported effect (Theory) for these news briefs. 

7. For all groups, the text dimensions of Typicality and Plausibility interacted. When presented with a news brief reporting a counterintuitive finding 
(Low Plausibility) within the natural and physical sciences (High Typicality), participants requested information about Theory at the expense of 
asking questions about Method. Perhaps participants had sufficient confidence in the methodology used by the researchers to suspend questions about 
method in favor of questions that would help explain, for example, how a poisonous gasoline additive could reduce air pollution (To see the full news 
brief, see Appendix). 



Table 7 

Mean Number of Questions as a Function of News Brief Type and Topic 





Low Plausibility 
Low Typicality 


Low Plausibility 
High Typicality 


High Plausibility 
Low Typicality 


High Plausibility 
High Typicality 


Social Context 


0.64 


0.65 


0.44 


0.47 


Agent/Theory 


1.06 


1.76 


0.82 


0.92 


Methods 


1.73 


0.61 


1.68 


1.61 


Data/Statistics 


0.59 


0.3 


0.34 


0.49 


Related Research 


0.22 


0 . 9 


0.53 


0.45 


Relevance 


0 . 1 


0.08 


0.05 


0.07 


Total # of Questions 


4.34 


4.3 


3.86 


3.52 



Educational and Scientific Importance 

Reading in science courses, both at the high school and university level, is often confined to textbooks that recount established scientific findings 
(Bauer, 1992). In this age of communication, however, information about cutting-edge science is readily available in the visual and print media, 
presenting numerous and continuous opportunities to learn about new scientific research. In fact, for most high school and university graduates, 
learning about new scientific developments occurs largely within informal contexts, such as through exposure to media reports. Consequently, 
competence in evaluating media reports of science is important not only for students planning to enter professions related to science, but for all high 
school graduates. To date, the effects of high school and postsccondary science education on evaluative competency are not well understood. In the 
present study, we identified the types of information individuals seek when evaluating scientific news reports, and we investigated how their 
questions are related to levels of formal education in science and text dimensions. 

We found that although all groups generated a variety of questions, the types of questions generated were constrained by the text dimensions of 
Plausibility and Typicality. We also found that high school science courses help to prepare individuals to think about some, but certainly not all, 
types of questions important for evaluating research. In fact, only questions about Theory were asked frequently by all groups. Frequent questions 
about other topics of research, such as Method, for example, were evident only in university students who were in the fourth year of their science 
program in Psychology. As our results suggest, however, even an undergraduate degree does not guarantee that individuals will think readily about 
all potentially important aspects of science when evaluating science news briefs (e.g.. Data, Related Research). Given the importance of informal 
learning about science for effective citizenship, science educators at all educational levels must ask themselves how to help students think critically 
and broadly about all aspects of science that arc important for assessing the significance of new scientific advances. 
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Appendix- News Briefs 



(Low Plausibility, Low Typicality) 



People have long been interested in what the future holds. Researchers have reported that people who wear Ollinite crystals during sleep are more 
likely to have dreams that predict the future. Members of Mind Matters have hailed this finding and have concluded that wearing this crystal Js 
important for increasing the frequency of dreams about future events. 



People are concerned about the environmental effects of automobile emissions. Researchers have reported that vehicles that bura gasoline containing 
the poisonous chemical Quipmanol will reduce existing levels of air pollution. Members of Antos for the Future have hailed this finding and 
concluded that fueling vehicles with gasoline containing this poisonous chemical is important ior decreasin g current Jeyels _of _ajr pollution. 



(High Plausibility Low Typicality) 



People in western countries have long been fascinated by traditional Eastern religious practices. Researchers have reported that senior citizens who 
practice Mai handu meditation show an increased sense of well being. Members of Lifestyles for Seniors have hailed this finding and have concluded 
that practicing this meditation is important for increasing the sense of well being in seniors. 



People arc concerned that declines in wildlife populations will result in extinction for some species. Researchers have reported that robins that have 
been exposed to the insecticide Permaldrin are less likely to mate than usual. Members of Nature Unlimited have hailed this finding and have 
concluded that using this insecticide is an important factor in causing a decline in robinsi mating behavior. 



(Low Plausibility, High Typicality) 



(High Plausibility, High Typicality) 
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